Learning Sentence-Level Representations with Predictive Coding

Learning sentence representations is an essential and challenging topic in the deep learning natural language processing communities. Recent methods pre-train big models on a massive text corpus, focusing mainly representation of contextualized words. As result, these cannot generate informative embeddings since they do not explicitly exploit structure discourse relationships existing contiguous sentences. Drawing inspiration from human processing, this work explores how to improve sentence-level pre-trained by borrowing ideas predictive coding theory. Specifically, we extend BERT-style with bottom-up top-down computation predict future sentences latent space at each intermediate layer networks. We conduct extensive experimentation various benchmarks for English Spanish languages, designed assess sentence- discourse-level pragmatics-focused assessments. Our results show that our approach improves consistently both languages. Furthermore, experiments also indicate capture pragmatics knowledge. In addition, validate proposed method, carried out ablation study qualitative which verified mechanism helps quality representations.